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1 INTRODUCTION 


1.1 Object 


This document is a presentation of D’Fusion® Augmented Reality and an introduction to all the basic concepts 
necessary to build an AR scenario. 


1.2 External documentation 


1.2.1 Reference documents 
[01] D’Fusion Studio - User Guide, Total Immersion 
DFusion Studio - User Guide.pdf 


[02] D’Fusion Augmented Reality - Reference Manual, Total Immersion 
DFusion AR - Reference Manual.pdf 


[03] D’Fusion Augmented Reality - Lua API, Total Immersion 
DFusion AR - Lua API.pdf 


[04] D’Fusion Augmented Reality - Physics Plugin, Total Immersion 
DFusion AR - Physics Plugin.pdf 


[05] D’Fusion Computer Vision - Reference Manual, Total Immersion 
DFusion CV - Reference Manual.pdf 


[06] D’Fusion Exporter for Maya - User Manual, Total Immersion 
DFusion Exporter for Maya - User Guide.pdf 


[07] D’Fusion Exporter for Maya - Modeling Constraints, Total Immersion 
DFusion Exporter for Maya - Modeling Constraints.pdf 


[08] D’Fusion Exporter for 3dsMax - User manual, Total Immersion 
DFusion Exporter for 3dsMax - User Guide.pdf 


[09] D’Fusion Exporter for 3dsMax - Modeling Constraints, Total Immersion 
DFusion Exporter for 3dsMax - Modeling Constraints.pdf 


[11] D’Fusion Computer Vision — Marker Tracking User Guide, Total Immersion 
DFusion CV — Marker Tracking User Guide.pdf 


1.2.2 Other documents 


[10] LUA 5.1 Reference Manual 
www.lua.org/manual/5.1 
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1.3 Glossary & Acronyms 


TI Total Immersion 

AR Augmented Reality 
MLT Marker Less Tracking 
CV Computer Vision 


1.4 Presentation rules 


Presentation | Content 
XXXXXX File name 
XXX | Script : code samples 
XXXXXXK Key word 
ll =XXXXXXX | Reminder : main information of the chapter 
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2 WHAT IS D’FUSION® AUGMENTED REALITY 


2.1 Augmented Reality 


Augmented Reality is a term used to designate the integration of 3D objects (and computer-generated data in 
general) into live video. With the help of Augmented Reality technology, the video is digitally processed, analyzed 
and “augmented” with the 3D components. 


In other words, Augmented Reality mixes real and virtual words together, in real time. 


Figure 1: Augmented Reality 


2.2 D’Fusion® AR Tools 


D’Fusion® Augmented Reality (D’Fusion AR for short) consists of several Augmented Reality tools within a 
framework designed to build Augmented Reality scenarios. 


These tools can be thought of as modules capable of communicating with each other within D’Fusion AR: 


e Video Capture module 


This is the main component of D’Fusion AR, dedicated to the management of live and prerecorded video 


captures. Notice it is not mandatory and you might create standard 3D applications with D’Fusion® 
Augmented Reality. 


e Real-Time Rendering Engine 


3D real-time rendering based on the Ogre 3D engine, essential for the development of Augmented Reality 
applications; but you can also embed sound, video files, images, etc. 


e Lua Scripting module 
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Allows advanced AR scripting based on the Lua scripting language. 


¢ Computer Vision module 
This part manages all the computer vision: 
e = Tracking 
Augmented Reality is built in the interaction of 3D with the real world. The main value thus relies 
on the integration of real time inputs, like position or orientation extracted by captors or devices. 
D’Fusion® Augmented Reality embeds a live position and orientation tracker: D’Fusion® 


Computer Vision (Marker Less Tracking); but you can also integrate specific tracker or hardware 
devices to match your needs. 


e Image Processing 
D’Fusion® Computer Vision provides a computer vision library that allows the user to achieve 
basic or advanced processing on the images retrieved by the camera. This library includes diverse 


object tracking or interaction algorithms (as Marker Less Tracking, Recognition or background 
subtraction) as well as basic image processing functions. 


e Natural Interactions 


The following graph shows the global D’Fusion AR architecture: 


Content Production: D’Fusion Studio Suite 


Media Production 
Hardware Scripting ‘D Fusion Exporiet fot Maya’. Specific Development 
. . sion Ex, fer for Is Max 
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Figure 2: D'Fusion® Augmented Reality Architecture 
None of these assets are mandatory, and you can always build simpler applications depending on your needs. 
On top of these modules, you will also find: 


e The D’Fusion® Augmented Reality SDK 
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This is the core of the engine under which all applications are built. The SDK offers flexibility and the 
possibility to always enhance your applications with homemade features. 


e The D’Fusion® Studio authoring software 
The authoring tool used to design and develop AR scenarios for deployment. 
e The D’Fusion® @Home and Mobile deployment tools 


This is a basic player build on the SDK, which enables the use of all the standard features exposed by the 
SDK. 


All D’Fusion Studio Suite components are built upon D’Fusion Augmented Reality. 
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3 D’FUSION® AR ENGINE: HOW IT WORKS 


D’Fusion® Augmented Reality is a scenario scripting engine for Augmented Reality applications. It is a real-time 
engine, giving access to a set of objects organized in “scenes”. Broadly speaking, one scene represents the 
execution of one AR scenario. 


This architecture is embedded in the D’Fusion® AR SDK, which is one layer directly executed in the D’Fusion® 
runtimes and accessible through plugins. 


SCRIPTING 
ENGINE 


SCENETTES 


SDK 


Figure 3: D’Fusion® Augmented Reality Engine 


You will see later that scenes are a quite modular notion, allowing advanced developers to create quite complex 
scenarios very efficiently. All the objects present in a scene, and the scene itself, are behavioral objects for the 
engine as almost all of them can be scripted. The notion of script is the unitary notion of behavior in the engine. 


Along with this behavioral objects, D’Fusion® Augmented Reality gives you access to a small set of tools designed 
to manage efficiently your behaviors and to interact dynamically within your scenario. These tools are the core of 
the engine: they are timelines and events. 


In the following documentation, all these objects will be described more in depth. You will see that, as most parts 
of the engine are very accessible to the user, but still simply packaged, you can create very basic Augmented 
Reality applications very fast but also create very complex scenarios with the same engine. 
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3.1 Lua Scripting 


3.1.1 Frame 


As D’Fusion® Augmented Reality is a real-time engine, it must control very precisely its process loop, and more 
specifically the scheduling and prioritizing of all tasks. Thus, the process loop is very important for the 
understanding of how the engine works. 


This loop, hereafter called the “main loop”, is embedded in one single process and thread. One loop is spitted in a 
defined sequence of steps, always being called in the same order on every cycle of the loop. One cycle of the loop 
is called a frame, and can be used as a time unit for behavior implementations. 


The following figure shows the execution of one frame with corresponding events. 


E 
VENTS 

P 

P 

P 

P 
OSTRENDE 
R 


Figure 4: D’Fusion® Augmented Reality Execution Steps 


This context of execution is very basic but also very important. Most real-time constraints come from this frame 
execution loop, as every single behavior which is processed must be designed as to not slow down the whole 
engine. Thus, any time consuming operation, such as tracking, should be threaded and managed explicitly to 
synchronize with the main loop. Time management will be a very important issue here. 


In the engine, the whole frame processing is synchronous. Whereas some process or plugins may need to be 
threaded (sound management, video buffering, CV processing for instance), all calls made in this main loop are 
synchronous so synchronization with these threaded must be dealt with during these calls. 


3.1.2 Behavioral Objects and Scripts 


As already mentioned, the atomic notion in the engine is the notion of behavioral object. This notion actually 
refers to objects which can be scripted. You can attach a script to almost every single object in the engine, 
making it very customizable. Moreover, this approach is reflexive, as scripts themselves are behavioral objects 
that can be scripted. 


The engine embeds management of scripts and behavioral objects as generic data. Lua scripting is just a specific 
implementation of this generic notion for example, but you can create your own script using any logic underneath 
with the SDK. 


All you might want to remember for the moment is: everything is scriptable. 
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3.1.3 Time management 


Time representation and time management are a very central notion for Augmented Reality applications. There 
are actually several notions of time which you might want to consider depending on your needs: 


e Real time 
This is the raw value of time, which is computed from the start of your application. This notion is a very 
common one, and present in any real-time engine. In the case of an Augmented Reality application, it 
might not be the more useful notion though 

e Frame time 


Most of the time, one video input will determine the whole process. The video is actually not played 
exactly in real-time because we have to compute information from the inputs before using it. To have 
smooth behaviors, and a good synchronization between what you see and how the engine reacts, this is 
then the time of the video which will be used as a global reference. This is the frame time. 


Both notions can also be represented in two ways: the standard notion of time, expressed in seconds, or the 
notion of time code. When the engine is to be determined by a video input, you can specify to the engine that 
this video input is the time “master”. 


3.1.4 Timelines 


A Timeline is a basic representation of behavior scenarisation through time. You can see it as a global manager 
for script processing. Any script which might be executed will be attached to a timeline before being processed. 


There is always one global timeline in a scene. By default, this timeline is the one used whenever you manipulate 
script. But you can also create and manage yourself additional timelines if needed. 


Depending on your needs, you can plan the execution of a script in 3 different ways: 


e Schedule the script 
This will for instance be the most commonly used way to create static scenarios. When you know the 
exact time when to trigger a behavior, you can just schedule it with absolute value of time. 

e Delay the script 
This will be more useful way for dynamic scripting: when you need to trigger a script in a given amount of 
time, you will “delay” it by n seconds, meaning the script will be processed after n seconds from now. 

e Trigger the script 


If you need to execute a script right now, without consideration of time issues, you can simply trigger it. 
Notice that a script can trigger itself, but it will in fact trigger it from the global scene timeline (it is just a 
shortcut). When triggering a script, the default behavior is to execute it at the next frame, but you can 
override this setting to execute it immediately if needed. 


REMINDER 


{Every object in the engine is scriptable 


{&l Use Timelines if you need to control timing of execution of your scripts 
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3.1.4.1 Lua good practices 


While Lua, as a script language, is very flexible, different syntaxes can lead to very different internal behaviors, 
and vast performances differences. This section will explain some of the important aspects of the Augmented 
Reality implementation of Lua and some good practices to keep optimal performances and memory use. 


3.1.4.2 _Coroutine.yield() 


coroutine.yield() is a call which will interrupt the execution of a script, so that is is started again the following 
frame. 


local a =1 
coroutine. yield() 
a=5 
repeat 
a=atl 
until coroutine. yield() 


Consider the following script sample: 


Execution of the script will proceed as follow: 


e during the first frame, the local variable ‘a’ will be created, and assigned the value 1. The call to 
coroutine.yield will then interrupt the script, and return execution to the D’Fusion® Augmented Reality 2 
engine. 


e during the second frame, the execution will resume at the coroutine.yield() call. So next instruction 
executed will be a=5. The script will then enter the repeat... until loop. ‘a’ will be incremented 
immediately again, so now, a = 6. The “until” condition will then be evaluated, which causes a call to 
couroutine.yield, so the thread will be interrupted again and the frame processing will be handed back to 
the Augmented Reality 2 engine. 


e the following frame (and all further ones after this one) will resume with the “until” condition being 
evaluated. To be precise, a call to coroutine.yield interrupts a script, and the script will resume with 
coroutine.yield returning. In D’Fusion® Augmented Reality 2, coroutine.yield return always tests as false. 
So such a way to write your repeat...until loop means you will never exit said loop, and is valid for scripts 
which should always be kept running. 


local a =1 
local exit = false; 
repeat 
coroutine. yiela() 
a=atl 
If (condition) then 
exit=true 
end 
until exit 


For scripts in which you need to be able to control and end the script, you can just choose the exit condition of 
your choice, and manually add the coroutine.yield call inside your loop: 
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Here, the script will be executed each frame, until the condition ‘condition’ (which can be a test performed in the 
loop itself, a global boolean modified in another script...) becomes true. This will in turn make exit become true, 
and the script will end. 


Remember: as the engine is not threaded, you must never create infinite loop inside your scripts without 
coroutine.yield() instructions, as the script would then never interrupt and the rendering or other process will 
never occur ! 


So the main points to remember: 


e coroutine.yield() interrupts a script execution for this frame, and set it to be resumed the following frame 


e scripts are not threaded : the script will keep on running until it reaches its end of file or coroutine.yield() 
is called, interrupting the main engine loop during all this time 


e in D’Fusion® Augmented Reality, the return of coroutine.yield() always tests as false. 


3.1.4.3 Variables allocation in Lua 


Variable allocation in Lua can be quite expensive, especially for variables which are not from a basic Lua type, but 
are bound from C++. This is mostly due to the overhead created by the binding mechanism. So you should 
always try to minimize the number of variable creation and destruction by paying attention to your variable 
scopes. 


Let’s take the example of the following Lua script, and how we can improve it by better management of our 
variables. 
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MLTPlugin = getMLTPluginManager() 
err_ret, trackingindex = MLTPlugin:startTracking(". ||scenario|\tracker.xml/", 0) 
coroutine. yield() 


repeat 
err_ret,status = MLTPlugin:getTrackingStatus(trackingindex) 


if(status == 1) then 
err_ret, targetcount = MLTPlugin:getTargetCount(trackingindex) 


for i=1,targetcount do 
scene = getCurrentScene() 
camera = Object3D(scene.getObjectByName("camera")) 
ref = Object3D(scene.getObjectByName("Ref")) 


err_ret, targetstatus = MLTPlugin:getTargetStatus(trackingindex, i-1) 
if(targetstatus==1) then 
position = Vector3() 
orientation = Quaternion() 
ref:setVisible(true) 
err_ret = MLTPlugin:getTargetPos(trackingindex, i-1, position, orientation) 
ref:setPosition(position, camera) 
ref:setOrientation(orientation, camera) 


else 
ref:setVisible(false) 
end --if 
end --for 
end --if 
until coroutine. yield() 


There are two main issues with this script. The first one is that all variables are created as global: in D’Fusion® 
Augmented Reality 2, all scripts share the same global common space. In Lua, variables are by default declared in 
this global namespace. This has quite a few inconvenient, mostly that it leads to quickly cluttering that 
namespace, and that the management of those variables is much less efficient than local variables. 
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local MLTPlugin = getMLTPluginManager() 

local err_ret = eOk 

local trackingindex = -1 

err_ret, trackingIndex = MLTPlugin:startTracking(". ||scenario||tracker.xm!/", 0) 
coroutine. yield() 


repeat 


local status = O 
err_ret,status = MLTPlugin:getTrackingStatus(trackingindex) 
if(status == 1) then 

local targetcount = O 

err_ret, targetcount = MLTPlugin:getTargetCount(trackingindex) 


for i=1,targetcount do 
local scene = getCurrentScene() 
local camera = Object3D(scene:getObjectByName(‘camera")) 
local ref = Object3D(scene.getObjectByName("Ref")) 


local targetstatus = O 
err_ret, targetstatus = MLTPlugin:getTargetStatus(trackingindex, i-1) 
if(targetstatus==1) then 
local position = Vector3() 
local orientation = Quaternion() 
ref:setVisible(true) 
err_ret = MLTPlugin:getTargetPos(trackingindex, i-1, position, orientation) 
ref:setPosition(position, camera) 
ref:setOrientation(orientation, camera) 


else 
ref:setVisible(false) 
end --if 


end --for 
end --if 
until coroutine. yield() 


Since this script does not need to share variables with other scripts which should be the only case where you 
want global variables, let’s re-declare all its variables as local: 


This script now makes sure not to clutter the global namespace, but creates another issue 
variables only exist in the scope they were created in, we will be constantly creating and redestroying variables, 
which in Lua has a cost at creation because of the overhead, and a cost in memory management, while variables 
not referenced anymore wait to be processed by the garbage collector. 
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local MLTPlugin = getMLTPluginManager() 

local scene = getCurrentScene() 

local camera = Object3D(scene:getObjectByName("camera")) 
local ref = Object3D(scene.getObjectByName("Ref")) 


local position = Vector3() 
local orientation = Quaternion() 


local err_ret = eOk 
local trackingindex = -1 
local targetcount = O 
local targetstatus = -1 
local i =0 
TrackingStatus = O 


err_ret, trackingindex = MLTPlugin:startTracking(’. | |scenario|\tracker.xml/", 0) 
coroutine. yield() 
err_ret, targetcount = MLTPlugin:getTargetCount(trackingindex) 


repeat 
err_ret, TrackingStatus = MLTPlugin.getTrackingStatus(trackingindex) 
if (TrackingStatus ==1) then 
for (=1,targetcount do 
err_ret, targetstatus = MLTPlugin:getTargetStatus(trackingindex, i-1) 
if(targetstatus = 1) then 
ref:setVisible(true); 
err_ret = MLTPlugin:getTargetPos(trackingindex, i-1, position, orientation) 
ref:setPosition(position, camera) 
ref:setOrientation(orientation, camera) 
else 
ref:setVisible(false); 
end --if 
end --for 
end --if 
until coroutine. yield() 


So as much as possible, variables should be reused. A final version of the script would now be 


Here, we ensure that we do not needlessly call at each frame D’Fusion® Augmented Reality 2 objects or 
managers which will not vary during our script execution. 


The two main points to remember: 


e do not use global variables if you do not need to share that variable. 


e try to limit the creation/destruction of variables, take those out of your main loop when possible. 
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3.2 Image Processing and Computer Vision 


3.2.1 D’Fusion® Computer Vision 


Computer Vision is the essence of Augmented Reality. It thus plays a central role in D’Fusion® Augmented Reality 
built applications. To ensure robustness, the engine does not embed computer vision behavior though. The issue 
is addressed through the D’Fusion® Computer Vision Plugin, which embeds all the computer vision possibilities 
from D’Fusion® (which means you could consider using D’Fusion® Augmented Reality without this plugin to create 
standard 3D applications without computer vision functionalities). 


D’Fusion® Computer Vision includes a set of image processing and computer vision functionalities as: 
e Basics Image Processing functions 
e Motion Detection 
e Background subtraction 
e Color object 2D tracking 
e Simple 2D multi Face and Eye Detection 
And 


e Marker Less Tracking and Recognition 


e Marker Tracking (pre-defined markers) 


3.2.1.1 Image Processing 


The “Image” Lua library includes basic image processing functions such as color conversation and image 
subtraction of image threshold. 


High level computer vision features are also included. For now, Motion Detection, Background subtraction, Color 
Object Tracking and Face Detection are bound. 


Hereunder is a brief description of these features: 


3.2.1,1.1 Motion Detection 


This Algorithm allows detecting moving parts in the video image. It uses the history of the subtraction of two 
consecutive images. 


This feature can be used for multiple kinds of interactions, for example you can trigger an action when somebody 
moves his hands in front of a defined area of the video image. You have to keep in mind that the detection is 
possible only if the objects (i.e. the hands) are moving into the image. 


3.2.1.1.2 Background Subtraction 


This algorithm allows extracting objects (from the video image) that do not belong to previously learnt 
background. It can be seen as complex image subtraction (subtraction between the current image and the image 
of the background). However, the learning of the background takes many from in order to improve robustness to 
noise and little changes in the background (it is not robust to lighting changes and cast shadow). 


This feature can be used for multiple kinds of interactions. In comparison with the Motion Detection, the objects 
can be detected even if there are no moving parts in the video image. You can for example interact with the 
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objects contours or find the object centers. You can use this feature as a generic chromakey too. However, the 
quality of the cut out shapes will not be as good as a real chromakey. 


3.2.1.1.3 Colored Object Tracking 


This algorithm allows tracking uniformly-colored objects. It is based on color histogram segmentation and mean 
shift tracking. The color of the object to track has to be previously learnt. This algorithm is not robust to 
automatic white balance compensation and large lighting changes (which can be due to automatic exposure 
setting of the camera). 


This feature can be used, for example, to track colored circles moving on a table or on a “Flipboard” which is 
tracked with the Marker Less Tracking. 


3.2.1.1.4 Simple 2D multi face and Eye Detection 


This algorithm allows detecting multiple faces in the same image. It detects the eyes when the face is large 
enough in the image. 


3.2.1.2 Marker Less Tracking 


Marker Less Tracking is part of D’Fusion® Computer Vision. 


|" 


This is a standalone engine based on a patented innovative technology dedicated to retrieve “natural” information 
provided by various real objects found in an image or in a video stream. Once these natural data is collected, 
objects can be identified and followed through consecutive images extracted from the video stream in real-time. 
This process is called “tracking”. 


To build an application using the D’Fusion® Computer Vision technology, you will need a good understanding of 
how to prepare the tracking data, you should refer to the D’Fusion® Computer Vision documentation [05]. 


3.2.1.3 Marker Tracking 


The Total Immersion patented technology D’Fusion Marker Tracking, is part of the D’Fusion Computer Vision 
Suite. 


The Marker Tracking is to be used in Lua with the plugin called “D’Fusion Computer Vision — Marker Tracking 
Plugin”. 


For more details, please refer to the D’Fusion Computer Vision — Marker Tracking User Guide [11]. 
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3.2.2 D’Fusion® Computer Vision Plugins 


The D’Fusion® Computer Vision Plugins makes the link between trackings and D’Fusion® Augmented Reality 
engine. The plugins essentially comes through a set of built-in functionalities exposed to Lua. 


3.2.2.1 Image Processing with Lua 


Except the Marker Less Tracking and the Recognition, Image Processing and computer vision functionalities are 
accessible in the script through an object called “Image” and objects derived from “Image”: 


e “Image”, for Basic Image Processing 
e “ImageBGSubtraction”, for Background Subtraction, derived from “Image” 
e “ImageColorObjectTracking”, for Color Object Tracking, derived from “Image” 
e “ImageFaceDetHaar”, for Multi Face Detection, derived from “Image” 
All the Functions of “Image” are accessible through the derived objects. 
Examples: 


To achieve an image conversion from a color space (i.e. RGB) to another (i.e. YUV): 


-- Image creations Width Height Bit per channel Number of channels 
local ImageRGB = Image( 640, 480, & ay 
local ImageYUV = Image( 640, 480, & BE 


-- Get RGB image from video capture 
ImageRGB.getFromCapture(capture); 


-- Color Conversion Destination image Convestion code 
ImageRGB.convertColor( Image YUV, 36 ); 


Face Detection: 


-- creation of face detection Image 


local ImageFaceDet = ImageFaceDetHaar( 320, -- Width 
240, -- Height 
3 -- Number of channels 


MaxFaceNumber ); -- Max nb of faces 
-- set face Classifier path 
local FaceClassifierPath convertString TOUTF&(‘'Classifiers/haarcascade_frontalface_alt2.xm!/"); 
-- Number of faces detected 
local FaceNumber = 0; 
-- Array of face parameters 
local FaceParameter = {}; 
-- load Classifiers that contain the features of the face 
imageFaceDet:LoadClassifiers( FaceClassifierPath ); 
-- Get RGB image from video capture 
ImageFaceDet.:getFromCapture(capture); 
-- detect faces 
FaceNumber, FaceParameters = imagefaceDet:DetectFaces(); 


For more information about the use of these functions, please refer to D’Fusion AR - Lua API documentation [03]. 


Examples are available in the “How To” section below. 


IMMERSION ®@ +33 (0) 146 25 97 42 


& www.t-immersion.com 


D’Fusion® Augmented Reality 


TOTAL IMMERSION a ” eas 15/10/12 


3.2.2.2 Marker Less Tracking with Lua 


You can control your tracking scenario and compute the results directly in your scripts. All these functions are 
exposed through a single Manager object. 


You can retrieve this object through the global function: 


MLTPlugin = getMLTPluginManager() | 


We suggest declaring a single global object in Lua, with this initialization in a script dedicated to the management 
of the plugin. You can however retrieve the object separately in different scripts if needed for some reason. 


All the methods to manage the tracking are exposed through this object; you can consult the Lua API 
documentation [03] for an exhaustive list and a description of these methods. 


Processing Data: 


Let’s now consider your tracking scenario is ready, and you want to use it within the plugin. You first need to start 
the tracking scenario: 


myVideo = VideoCapture(scene.getObjectByName('myLiveCapture*); 


trackingindex = MLTPlugin:startTracking("TrackingScenario.xml", myVideo:getVidCapID()), 


As mentioned, the « trackingScenario.xml» is the result of preparing your tracking scenario with D’Fusion® 
Studio Computer Vision. We consider you also created a VideoCapture object in your project called here 
“myLiveCapture” which is the stream where you want tracking to be done. 


The startTracking method also needs the ID of your videoCapture. We strongly recommend not to use directly 
the ID you think might be attributed, but always dynamically retrieve it to avoid mistakes. 


This method finally takes an optional parameter, which is the camera. If set, it will override the calibration values 
of the camera used by D’Fusion® Computer Vision for the tracking. 


Once started, the data is automatically processed by the D’Fusion® Computer Vision plugin in the video you 
specified as input. 


There are 2 important notions you should be aware of when retrieving the data from the plugin: 


e All the data is synchronized with the rendering of the video 


There might be delays between the capture and the rendering in your video stream; the output of the 
plugin corresponds to the current time of the application, driven by the master video input, so you should 
make sure your input video is always master for a correct synchronization. 


e All the data is computed relatively to the current camera 


Positions and orientations computed in the D’Fusion® Computer Vision plugin cannot be abstract values, 
so we always make the assumption that it is relative to the current camera of the scene, with the input 
video rendered as background. If you are dealing with other cases in your scenarios, you may need to 
make all the necessary 3D transformations to match your configuration. 


RIEIMiTNID) EIR 


D’Fusion® Computer Vision Marker Less Tracking scenario must be prepared with the D’Fusion® Studio 


fel 


Computer Vision, which creates a .xml file embedding the scenario 


(al 


D’Fusion® Computer Vision Marker Less Tracking Plugin returns values relative to the current camera 


te! 


D’Fusion® Computer Vision Marker Less Tracking Plugin manages synchronisation with the live Video 


Input 
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3.2.2.3 Marker Tracking with Lua 


You can control your marker tracking scenario and compute the results directly in your scripts. All these functions 
are exposed through a single Manager object. 


You can retrieve this object through the global function: 


MarkerPlugin = getMarkerPluginManager() | 


For more details, please refer to the D’Fusion Computer Vision — Marker Tracking User Guide [11]. 


REMIT NiDIEIR! 


& D’Fusion® Computer Vision Marker Tracking Plugin returns values relative to the current camera 
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4 OBJECTS DESCRIPTION 


Here is a presentation of all objects defined within the D’Fusion® Augmented Reality framework which you can 
manipulate. All the parameters exposed here can be used directly in the definition of objects in your scene. 


There is also a clear class hierarchy, so any parameter defined in a parent class is of course accessible through all 
derived class. 


For more information on SDK and Lua scripts methods, please refer to the relevant documentation. 


4.1 Object 


This is the base class for any object dealt with by the engine. It embeds the management of scripts, and internal 
methods to manage all the objects smoothly in scenes and modules. 


It is an abstract class though, and you shall not instantiate it directly. 


Object -Parameters 


Parameter : ‘name’ Type: ‘string’ 


Every object in the engine has a unique name. This name is also an identifier you can use to retrieve this object in your 
scripts afterwards. It is a good habit to give all your objects specific names. If no name is provided, the object will 
always be assigned a generic name on the form “classKXX” where class is the name of the object class, and XX the 
lowest available number. 

Besides if the name you provide is not available, the object will also be renamed automatically by adding a number 
suffix. 


Parameter : ‘file’ Type : ‘string’ 

Most objects may have resources you need to load to instantiate it. For a generic use, the notion of file is thus managed 
at this level in the engine. There are restrictions on how resources can be managed by the engine, please refer to 
corresponding documentation for more information. 


CAUTION : Concerning Ogre media (mesh files, materials files, texture files, etc.), every file name must be 
unique. If the same file name appears in different folders of the scenario, there can be unexpected behaviors. 
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4.2 Object3D 


Derives from: Object 
Attachable to: Object3D 


This is the basic notion of 3D manipulable object. Taken alone, we could consider it as a 3D node in the scene 
graph, as the methods exposed by this object are all those of a 3D node: 3D moves and hierarchy management. 


It is also the base class for all specific objects which derive from it, which is why we generically called it 
Object3D: Lights, Cameras, Entities... all derives from Object3D for example. 


Object3D - Parameters 


Parameter : ‘position’ Type: “Vector3D’ default value: “{0,0,0}” 


The standard 3D position information of an object. The Vector is filled with format “{x, y, z}”. The information is 
always local, ie relative to parent if the object is attached to another 3D object, and world position if attached to the 
scene root directly. 


Parameter: ‘orientation’ Type: ‘Euler angle’ or‘Quaternion’ default value: “{0,0,0}” 


Just like position, orientation is set as relative to parent in project files. You can provide either Euler angle or 
quaternion. 


Parameter : ‘scale’ Type: “Vector3D’ or ‘double’ default value: “1” 


You can apply scales on your nodes directly in project files. The scale can be uniform if you just give a scalar, or 
different on each axis. Be aware of Ogre behavior with non uniform scales, please refer to the media documentation. 


Parameter : ‘visible’ Type : ‘boolean’ default value: “true” 


You can modify object visibility on start-up with this parameter. Notice that visibility is always inherited from parents, 
meaning that if an object3D is not visible, all children will also be invisible in the scene, no matter what their own 
setting is. 

In other words, an object is visible if its visibility flag is true, and if all its parents are visible. 


Parameter: “rendergroup’ Type: ‘uint’ in range [0,105] default value: “50” 


This is the rendering option that let you modify rendering priorities. Higher values are used for foreground object, 
lower values for background. Object in the same group are of course rendered using the usual Z-order sorting. This 
parameter has effect only on Entity and Particles. 


Parameter : ‘lookat’ Type : “Vector3D’ 


The ‘lookat’ parameter is an option to sets object orientation not directly, but depending on a constraint. If present, this 
information will thus override the orientation parameter, so that current —Z vector of the object is aligned to “look at” 
position given. 
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4.3 Entity 


Derives from: Object3D 
Attachable to: Object3D 


Entities are the physical 3D objects present in a scene, e.g. objects with a geometrical description (meshes). 
Natively, the engine deals with mesh files exported in Ogre format. These files usually can embed animation 
descriptions, skeletons, etc. The engine gives access to all the animation features that Ogre can deal with. 


Entity -Parameters 


Parameter : ‘file’ Type: ‘string’ 

The file parameter, as derived from Object, is used for entities to specify the mesh file. The .mesh file must be a valid 
Ogre object 

Parameter : “boundingbox’ Type : ‘boolean’ default value: “false” 


You can choose to display the bounding box of an entity by setting this parameter to true. 


Parameter : “cast shadows’ Type : ‘boolean’ default value: “true” 
Let you specify that this object should cast a shadow. 


TOTAL ee Charles de Gaulle DFusion AR - Reference Manual.doc 24/45 
IMMERSION ® +33 (0) 146 25 97 42 


& www.t-immersion.com 


D’Fusion® Augmented Reality 


TOTAL IMMERSION a ” eas 15/10/12 


4.4 Light 


Derives from: Object3D 
Attachable to: Object3D 


Lights are an important type of object in a 3D rendering operation, as a good tweaking of lights can really makes 
the difference in the final rendering quality of a scene, or in the other hand gives a really poor impression if badly 
designed. 


There are different types of lights: point, spot, or directional. There is also a global ambient light setting but this 
one is managed slightly differently by the rendering engine and thus not exposed as a light object. Many 
parameters are exposed, such as emissive or specular colors of course, but also shade management. 


Lights - Parameters 


Parameter : ‘type’ Type: ‘string’ default value: “point” 

There are actually 3 types of lights you can use: point, spot, or directional. Default is point, but you can modify the type 
with this parameter. Depending on the type chosen, following parameters may apply or not. 

Parameter : “enabled’ Type: ‘boolean’ default value: “true” 


You can disable a light at startup by setting this value to false. Lights are very CPU consuming, so it is very important 
to always optimize their uses. Use this setting to always disable lights that you don’t need to be applied on the 
beginning of your scenarios. 


Parameter : ‘diffuse’ Type : ‘Vector3D’ default value: “{1,1,1}” 


The default colour value of a light is full white. You can modify this color here if needed, with RGB values. 


Parameter : “specular’ Type: ‘Vector3D’ default value: “{0,0,0}” 


There is no specular colour on lights by default. Use this setting to apply a specular color with RGB values. 


Parameter: “innerspotrange’ Type: ‘double’ default value :— applies to: spot 
This setting only applies to spot lights. It sets the angle covered by the spot inner cone. 


Parameter: “outerspotrange’ Type: ‘double’ default value :— applies to: spot 
This setting only applies to spot lights. It sets the angle covered by the spot outer cone. 


Parameter : “spotfalloff’ Type : ‘double’ default value :— applies to: spot 
This setting only applies to spot lights. It sets the falloff between inner and outer cone of the spot light.. 


Parameter : ‘attenuation’ Type: ‘Vector4D’ default value :— applies to: all but 
directional 


By default, there is no attenuation on lights. You can however apply a standard attenuation computation on spot light 
and point lights (attenuation doesn’t make sense on a directional light which position is not relevant). 
There are 4 parameters to set : 

> range : the upper range of the light 

> 3 factors of attenuation, constant, linear and quadratic. 


Parameter : “cast shadows’ Type : ‘boolean’ default value : ‘false’ 
If set to true, shadows will be calculated from this light. 


Parameter : “shadowcamerasetup’ Type: ‘string’ 


When rendering texture shadows, you can specify a light to use a custom shadow camera with this setting. Possible 


values are : 
> default 
> focused 
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Lights - Parameters 
> focusedaggressive 
> lispsm 
> lispsmaggressive 


4.5 Camera 


Derives from: Object3D 
Attachable to: Object3D 


Cameras have a specific role in the engine: they are both a 3D object, with position and orientation, part of the 
scene graph, and thus can be manipulated as any other object. They also are the objects which might constraint 
the rendering operation. As the relationship between a real camera and the rendering made in the engine is very 
important for an augmented reality, the notion of “model” will be very important here. Note that a camera can be 
present in the scene graph without being used for a rendering operation, it will then just behave like a 3D basic 
object. 


All the standard camera notions are exposed here, like clipping planes, field of view, aspect ratio, etc. 


Camera - Parameters 


Parameter : ‘file’ Type: ‘string’ 


The file parameter, as derived from Object, is used in Camera to set the model of the camera. Models are XML files, 
with important information used to configure the rendering made from this camera so that virtual object and real world 
match perfectly when a live stream is used as background input. 

For more information about camera models, refer to the Computer Vision documentation 


Parameter : “nearclip’ Type : ‘double’ default value: “0.01” 


The nearclip of a camera is the distance under which object will not be rendered. 


Parameter : “farclip’ Type : ‘double’ default value: “10000.0” 


The farclip of a camera is the distance above which object will not be rendered, to optimize speed by not trying to 
render too small objects for example. 


Parameter : “fovy’ Type : ‘double’ default value: “0.785398” 


Let you override the fovY angle (in radians) of the camera. Be aware that this setting is constrained by the model of the 
camera if specified. If you set this parameter, it will thus override the model value. 


Parameter : “aspectratio’ Type : ‘double’ default value: “1.333” 


Let you override the aspect ratio of the camera. Be aware that this setting is constrained by the model of the camera if 
specified. If you set this parameter, it will thus override the model value. 
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4.6 Viewport 


Derives from: Object 
Attachable to: Scene, RenderTarget 


Viewports are the link between a camera and a render surface (RenderTarget). For this reason, they are quite 
particular objects in the engine, especially in the creation process. You must create a viewport on an existing 
RenderTarget, based on an existing camera. 


If the parent specified is a scene, the viewport will be created on the main RenderTarget (most of the time it’s 
the application RenderWindow) 


Viewport - Parameters 


Parameter : “camera’ Type : ‘Camera’ MANDATORY 


This field indicates with which camera the rendering will be made on this viewport. This field is mandatory, and the 
creation of the viewport will fail if not filled. 


Parameter : ‘dimensions’ Type: ‘vector4’ default value: ‘{0,0,1,1}’ 


Dimensions of the viewport in the render Target, as relative {top, left, with, height} values. 


Parameter : ‘background’ Type : ‘texture’ 


Specifies the texture used as background. The value must be the name of an existing texture. 


Parameter : ‘bgvideorect’ Type : ‘integer’ default value: *{0,0,1,1}’ 


Dimensions of the background as {top, left, width, height} values. Values are relative. 


Parameter: “backgroundcolor’ Type: ‘Vector4’ default value: ‘{0,0,0,1}’ 
Specifies the RBGA color value of the background. 


Parameter : “scheme Type: ‘string’ default value: “default” 


The notion of scheme is used by Ogre render engine to modulate rendering for a single material depending on context. 
Please refer to Ogre documentation for more information. 


Parameter: “orerenderscript’ Type: ‘script’ 


On rendering operations with different viewports, you may need to customize your scene for each single operation 
differently. For this purpose, you can attach a script which will be called before the rendering of each single viewport 
with this parameter. The parameter must be the name of an existing script 


Parameter: “postrenderscript’ Type: ‘script’ 


Like previous parameter, this let you customize your scene on rendering operation for each viewport. This script will be 
executed right after the rendering operation of the viewport, generally to restore things modified by the pre-render 


script. 
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4.7 RenderTarget 


Derives from: Object 
Attachable to: Scene, Module 


RenderTarget is the base class for managing rendering operations. Schematically, rendering is the operation of 
drawing all what is “viewed” (ie what is in the frustum) by a camera on a surface. The link between the surface 
and the camera is managed by viewports. RenderTarget is the basic notion which manages surfaces where you 
rendering can occur. There are in facts 2 main kinds of render targets: windows or textures. 


RenderTarget - Parameters 


Parameter : “custom Type : ‘bool’ default value: ‘false’ 


Generic render targets are managed by the engine in the common process loop. If you are using a specific render target 
for your own need, like a texture used in final rendering, you will need to set this parameter to true. This enables the use 
of most of following parameters 


Parameter : OIALOneiey Type 5 ‘integer’ default value: “0” 


requires ‘custom’ parameter to be true. 
When the render target is custom, you can specify a priority which indicates in which order all the custom render 
targets will be rendered, which might be necessary if somes use the result of some others. 


Parameter : “swapbuffers’ Type: ‘bool’ default value: “true” 


requires ‘custom’ parameter to be true. 
For custom render target, you can choose not to swap buffers automatically during update operations. This let you swap 
buffers manually when required. 


Parameter: “backgroundcolor’ Type: ‘vector3’ 


There is actually no such notion as background color on a render Target, since rendering occurs in viewport. We give 
you here thus a shortcut to apply an homogeneous background color on all viewports attached if required. 


Parameter: “orerenderscript’ Type: ‘script’ 


On rendering operations with different render target, you may need to customize your scene for each single operation 
differently. For this purpose, you can attach a script which will be called before the rendering of each single target with 
this parameter. The parameter must be the name of an existing script 


Parameter : “postrenderscript’ Type: ‘script’ 


Like previous parameter, this let you customize your scene on rendering operation for each target. This script will be 
executed right after the rendering operation of the render target, generally to restore things modified by the pre-render 
script. 


Parameter : “camera’ Type : ‘camera’ 


This is an optional parameter. Usually, you will attach viewports directly to the render target in your scene files. If you 
just want a single viewport covering the all target, you might create it directly with this parameter based on the camera 
given as parameter. (it must be the name of a valid camera) 
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4.8 RenderTexture 


Derives from: RenderTarget 
Attachable to: Scene 


RenderTextures are specific RenderTargets, embedding the management of intermediate rendering operations 
toward a texture which can be used in the scene. Several RenderTextures might be combined to produce the final 
rendering in order to achieve reflection and refraction effects for example. 


RenderTexture - Parameters 


Parameter : ‘width’ Type: ‘integer’ default value: ‘640’ 
Width of the texture. 


Parameter : “height’ Type: ‘integer’ default value: ‘480’ 
Height of the texture. 


Parameter : “fsaa’ Type: ‘integer’ default value: ‘0’ 


Level of multi-sample anti-aliasing used for this render texture. 


Parameter : “pixelformat’ Type: ‘integer’ 
Format used for the texture: PF_A1R5G5B5, PF_R8G8B8, PF_FLOAT32_RGBA, etc. 


4.9 Texture 


Derives from: Object 


Attachable to: none 


Texture - Parameters 


Parameter : “texturetype’ Type : ‘integer’ default value: ‘2d’ 


Type of texture. Admissible values are: 1d, 2d, 3d, cubemap 


Parameter : ‘nummipmap’ Type: ‘integer’ 


Sets the number of mipmaps to be used for this texture. 


Parameter : ‘gamma’ Type: ‘float’ default value: ‘1.0’ 


Sets the gamma adjustment factor applied to this texture on loading the data.. 


Parameter : “hawgamma’ Type : ‘boolean’ default value: ‘false’ 


Sets whether this texture will be set up so that on sampling it, hardware gamma correction is applied. This option is 
only supported on recent hardware. 


Parameter : “isalpha’ Type : ‘boolean’ default value: ‘false’ 


This parameter is used if a greyscale texture must be used as an alpha texture. 


Parameter : ‘pixelformat’ Type : ‘integer’ 
Format used for the texture: PF_A1R5G5B5, PF_R8G8B8, PF_FLOAT32_RGBA, etc. 


IMMERSION ® +33 (0) 1 46 25 97 42 


& www.t-immersion.com 


D’Fusion® Augmented Reality 


TOTAL IMMERSION a ” <@% 


15/10/12 
User Guide ey 


4.10 Material 


Derives from: Object 


Attachable to: none 


Materials loading operations is managed directly by Ogre engine, based on dependencies to other assets. Thus, 
you cannot yet create Materials directly in project data. 
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4.11 Overlay 


Overlays are objects used to manage 2D interface. Like for most rendering assets, we directly use the Ogre native 
overlay format. Please refer to Ogre documentation for more information about overlay files format. 


Overlay - Parameters 


Parameter : ‘file’ Type: ‘string’ 


Reference to the resource file of this overlay. It must be a valid ‘.overlay’ Ogre format file. 


Parameter : “overlayname’ Type: ‘string’ MANDATORY 


Name of the Overlay to instantiate in the ‘.overlay’ file. This field is mandatory and object creation will fail if not 
provided or if the name does not refer to a valid overlay. 


Parameter : “visible’ Type : ‘boolean’ default value: ‘true’ 


You can use this parameter to turn the overlay visible or invisible on startup. 


Parameter : “zorder’ Type: ‘integer’ default value: ‘100’ 

If you have several overlays, you can use this parameter to determine which one is displayed on top. Values between 0 
and 650 are valid. 

Parameter : ‘rotate’ Type : ‘double’ default value: ‘0’ 


Rotation of the overlay in degrees. Rotation’s center is overlay’s center. 


Parameter : “scalex’ Type: ‘double’ default value: ‘1.0’ 


You can use this parameter to scale the overlay. 


Parameter : “scaley’ Type: ‘double’ default value: ‘1.0’ 


You can use this parameter to scale the overlay 


Parameter : “scrollx’ Type: ‘double’ default value: ‘0’ 


You can use this parameter to move the overlay. Values are in relative mode (left side of ’’screen” : 0, right side : 1) 


Parameter : “scrolly’ Type : ‘double’ default value: ‘0’ 


You can use this parameter to move the overlay. Values are in relative mode (top of “screen”: 0, bottom : 1) 


Parameter : “copy’ Type : ‘boolean’ default value: ‘false’ 


If you create two overlays or more, with the same file/overlayname combination and copy=false, these overlays are 
linked (it means that if you scroll one overlay, the second is scrolled also) 
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4.12 Text2D 


Text2D are used to display easily text on screen. 


Text2D -Parameters 


Parameter : “value’ Type: ‘string’ default value: *’ 
Displayed text. 
Parameter : ‘metricsmode’ Type: ‘string’ default value: ‘relative’ 


Unit used for all numeric parameters. Possible values are ‘relative’ or ‘pixel’. In relative mode, values are between 0.0 
and 1.0. (left side of ’screen” : 0, right side : 1) 


Parameter : ‘left’ Type : ‘double’ default value: ‘0.0’ 


You can use this parameter to move the text on the screen. Values’ unit depend on ‘metricsmode’ parameter. 


Parameter : ‘top’ Type : ‘double’ default value: ‘0.0’ 


You can use this parameter to move the text on the screen. Values’ unit depend on ‘metricsmode’ parameter. 


Parameter : “charheight’ Type : ‘double’ default value: ‘0.020000’ 


Font’s size. . Values’ unit depends on ‘metricsmode’ parameter. Values’ unit depend on ‘metricsmode’ parameter 


Parameter : “spacewidth’ Type : ‘double’ default value: ‘0.017320’ 


Space between characters. Values’ unit depend on “‘metricsmode’ parameter 


Parameter : “fontname’ Type: ‘string’ MANDATORY 
Font used for displaying text. 


Parameter : “colour’ Type : ‘vector3’ default value: *{1,1,1}’ 


Text’s colour. 


Parameter : “colourbottom’ Type: ‘vector3’ default value: *{1,1,1}’ 


Color of the text’s bottom part. 


Parameter : ‘colourtop’ Type : ‘vector3’ default value: *{1,1,1}’ 


Color of the text’s top part. 


Parameter : “textalignment’ Type: ‘string’ default value: ‘left’ 


You can use this parameter to align the text. Possible values are ‘left’, ‘right’ and ‘center’. 


Parameter : “show’ Type : ‘boolean’ default value: ‘true’ 


You can use this parameter to turn the text visible or invisible on startup. 


Parameter : “visibilitymask’ Type: ‘integer’ default value: ‘-1’ 


Applies a visibility mask to this object. 
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4.13 Sound 


Derives from: Object3D 
Attachable to: Object3D 


Sounds are a very important part of the immersion you might feel when using a 3D engine application. The 
engine embeds the OpenAL implementation of sound rendering on @home and iOS plateforms, and java 
implementation on android devices. The sound objects are natively exposed as 3D objects to make it much more 
easy to use the spatialization effects. Note that you can always disable the spatialization effects on a sound, the 
positions and orientations will then just have no effect. 


The implementation also permits to add predefined EAX effects, which adds a new dimension in the experience. 


Specific constraint :; for sounds that must be recorded during video recording, they must be Mono, 16 bits, 
44100 Hz and the parameter “streaming” must be set to “true”. 


Sound - Parameters 


Parameter : ‘file’ Type : ‘string’ 


Refers to sound resource. We actually manage ‘.wav’ and ‘.ogg’ sound formats. 


Parameter : “spatialize’ Type : ‘boolean’ default value: ‘false’ 


Even though sounds are 3D objects, most uses of sound may not use this feature and may prefer the sound to play ina 
constant and homogeneous way no matter where it is located in the 3D space. If spatialization, ie modulation of sound 
intensity depending on the sound localization relatively to the current camera, is required, set this parameter to true. 


Parameter : “loop’ Type : ‘boolean’ default value: ‘false’ 


Set this parameter to true if you want the sound to loop when played. 


Parameter : ‘predefinedfx’ Type: ‘string’ 


The OpenAL implementation used allows use of some advanced reverberation or other kind of effects on sounds. A set 
of predefined effects can be used to simply add nice immersive impression in your scenario. Use this parameter to 
apply on of this effect to this sound. 

Admissible values are : 


> REVERB PRESET BATHROOM 

> REVERB _PRESET_ LIVINGROOM 

> REVERB PRESET _CONCERTHALL 

> REVERB _PRESET_HANGAR 

> REVERB _PRESET_FOREST 

> REVERB PRESET _SPORT_SQUASHCOURT 

> REVERB PRESET _SPORT_GYMNASIUM 

> REVERB PRESET _OUTDOORS_VALLEY 

> REVERB _PRESET_CHAPEL 
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4.14 Particles 


Derives from: Object3D 
Attachable to: Object3D 


Particles are used to create effects like smoke, clouds, flames, fireworks.... A particle system is described in a 
particle script, as a material in a material script. 


D’Fusion® Augmented Reality offers two types of particle systems: ParticlesFX (Ogre particles plugin) and 
ParticlesPU (Particle Universe plugin for Ogre). 

4.14.1 ParticlesFX 
Derives from: Particles 


Attachable to: Object3D 


ParticlesFX —- Parameters 


Parameter : “file’ Type: ‘string’ 


Refers to particles resource (.particles file) 


Parameter : ‘particlesname’ Type : ‘string’ 


The name of the particles template in the .particles file. 


Parameter : ‘started’ Type : ‘boolean’ default value: ‘true’ 


Set this parameter to true if you want the particles to play when the scenario is launched. 
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4.15 VideoCapture 


Derives from: Object 
Attachable to: Scene 


VideoCapture objects are, of course, very important in an Augmented Reality application. The capture itself is 
exposed as an object as it permits fine control on inputs without interaction with how it is used afterwards; the 
only way to use a VideoCapture for rendering is through a VideoTexture object, but we can have many texture for 
one single capture, or even no texture at all. 


The VideoCapture notion will also play a very important role as it is a shared resource for the Total Immersion 
framework, and especially the D’Fusion® Computer Vision Plugin. You will find on capture object initialization 
methods, and methods to control the input, which will be different depending on if it is a live input or capture 
from a file. 


Sound - Parameters 


Parameter : ‘master’ Type : ‘boolean’ default value: ‘false’ 


Set this parameter to true if you want the video to control the time of the animations. 


Parameter : ‘loop’ Type : ‘boolean’ default value: ‘false’ 


Set this parameter to true if you want a video file to loop when played. 


Parameter : ‘autoplay’ Type : ‘boolean’ default value: ‘true’ 


If this parameter is set to true (which is default value), the video will automatically start playing on scene start event. 
Set this to false if you don’t want the capture to start. 


Parameter : “selectdialog’ Type : ‘boolean’ default value: ‘false’ 


If this parameter is set to true, and if the videocapture fails to open, then a selection dialog is displayed. 


Parameter : “compactdialog’ Type: ‘boolean’ default value: ‘true’ 


If this parameter is set to true, the selection dialog displays videocapture devices’ names only. If it is set to false, the 
dialog displays pixel formats and resolutions too. 


Parameter : ‘save’ Type : ‘boolean’ default value: ‘false’ 


If this parameter and selectdialog are set to true, then the videocapture which is successfully selected is saved as a new 
xml configuration. 


4.16 VideoTexture 


Derives from: Texture 
Attachable to: Scene 


As mentioned above, if you want a visualization of your VideoCapture in your scene, you will need to embed it in 
a VideoTexture. This object simply derives from a Texture object and embeds the VideoCapture. You may then 
have multiple VideoTexture embedding the same VideoCapture objects. 


VideoTexture - Parameters 
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VideoTexture - Parameters 


Parameter : “videocapture’ Type: ‘string’ 


The parameter must be the name of an existing videocapture 


Parameter : ‘gamma’ Type: ‘float’ default value: ‘0.0’ 


Sets the gamma adjustment factor applied to this texture on loading the data. 


4.17 Scene 


Derives from: Module 
Attachable to: Not attachable 


The scene is the main object for a project, corresponding to your top .dpd file. It thus holds most managers and 
is the main entry point for all access to sub-objects. There is only one scene in a project. 


Scene - Parameters 


Parameter : ‘ambient’ Type : ‘vector3’ 


Color of the ambient light 


Parameter: “shadowtechnique’ Type: ‘string’ default value: ‘none’ 


Shadow technique used for the scene : 
Admissible values are : 
stencilmodulative 
stenciladditive 
texturemodulative 
textureadditive 
textureadditiveintegrated 
texturemodulativeintegrated 


VVVVVV 


Parameter : “shadowtextureselfshadows’ Type: ‘boolean’ default value: ‘false’ 


Set this parameter to true if you want to force Ogre to apply shadow texture on casters. 


Parameter: “shadowcastermaterial’ Type: ‘string’ 


The shadow caster material is the default material applied on objects during the render of the shadow texture. 


Parameter : “shadowcasterrenderbackfaces Type: ‘boolean’ default value: ‘true’ 


By default, in texture based shadows, only back faces of casters are rendered 


Parameter : “shadowtexturesize Type: ‘integer’ 


Square size of the shadow texture. All shadow textures have the same size. 


Parameter : “shadowtexturecount’ Type: ‘integer’ 


The number of shadow textures. 


Parameter : “shadowtexturepixelformat’ Type: ‘string’ 


Pixel format of the shadow texture. For shadow-mapping, a value as pf_float16_r or pf_float32_r is advised. 


Parameter: “shadowfardistance’ Type: ‘double’ 


The “shadow far distance” is the distance from the light position at which shadows are terminated. 
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Parameter : “shadowcolour’ Type: ‘vector3’ 
This parameter is only valid for texture-based shadows. It allows coloring shadowed areas. 
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4.18 Luascript 


Derives from: Script (virtual) 
Attachable to: Object 


Luascript Objects are the native kind of script you can attach to object to manage your scenario, interactions and 
behaviors. They permit instanciation of script with the Lua Language integration of D’Fusion AR engine. 


The file parameter must references a valid .lua file for the object to be loaded correctly 


Luascript - Parameters 


Parameter : ‘trigger’ Type : ‘boolean’ default value: ‘false’ 


This parameter is a shortcut to force triggering execution of the script on launch of the scenario. 


Parameter : ‘priority’ Type : ‘integer’ default value: ‘0’ 


Priority manages order of execution of scripts. See script documentation in this manual for more information 


Parameter : “framerate’ Type : ‘integer’ default value: ‘-1’ 


A script can stay active over time in D’Fusion engine, using the yield() function. In the default case, it will thus be 
executed once for each frame. In some cases, the user might want the script to be executed at a lower framerate. This 
parameter is a helper to simulate a delay between 2 executions of the script. If you give a framerate of 2 for example, eg 
you would like the script to be executed 2 times per second, once the script is executed it will be planned for execution 
after a delay of 0.5 seconds on the timeline, instead of being planned on the very next frame. 

With the management of timeline, the resulting real delay will always be equal or superior to this delay, resulting in a 
real framerate always inferior or equal. Notice this method is just a helper, and the framerate resulting might not be 
constant. 


Parameter : “enabled’ Type : ‘boolean’ default value: ‘true’ 


If this parameter is set to false, the script will be disabled, preventing any execution before being re-enabled by the 
engine (via another script for example). 


Parameter : ‘paused’ Type : ‘boolean’ default value: ‘false’ 
This parameter is usefull only if the ‘trigger’ parameter is set to true. In this case, the script can be paused before being 
executed. 
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5 PROJECT FILE DESCRIPTION 


5.1 Format 


The contents of a project file (.dpd) resemble the following: 


<?xml version="1.0" standalone= "yes" ?> 
<dfusion version="1.0” > 
<settings> 
<rendering fullscreen='"false" vsync= "true" width="800" height="600" /> 
<application icon="Myicon.ico" title="My personalized player" /> 
</settings> 
<scene> 
<camera position="{0.0, 0.0, 0.0)" orientation="{0.0,0.0,0.0}" name="camera" /> 
<light position="{0.0, 0, 0.0)" name="lightO" type="point"/> 
<Videocapture name="vidCap1" file="video.xml" /> 
<videotexture name="background” videocapture="vidCap1" /> 
</uascript name="track" file="scenario.lua" trigger="true" /> 
<entity name=" glasses" file="glasses.mesh" scale="1.0" /> 
<viewport name="viewportO" camera= "camera" background= "background" /> 
</scene> 
</dfusion> 


This file is an XML file, with predefined tags. The loading of this file will always succeed as long as the XML format 
is valid and a D’Fusion® “scene” is present. This means that many unknown XML nodes can be present in the file, 
they will just not be processed (a warning message will be issued if log activated). 


You can notice the first tag in this file, “settings”. D’Fusion® Augmented Reality lets you dynamically embed a 
scene into others, which means that one scene is the main “application scene” while the others are kind of 
“subordinates”. This tag is detailed in next chapter. 
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5.2 Settings 


The settings tag contains information relative to the player configuration. This node only applies if present in the 
main scene file, and will be ignored if present in sub-scenes embedded in the main scene. 


Since these settings manage specific application values, there are some limitations in the way these sub-tags can 
be used however, and they differ from each other. 


5.2.1 Rendering 


The rendering tag is an important configuration node. It directly manages the application settings: 


rendering - Parameters 


Parameter : “fullscreen’ Type : ‘boolean’ default value: ‘false’ 

Set this parameter to true if you want your scene to start fullscreen. This setting is always efficient in the scene 
configuration. 

Parameter : ‘width’ Type : ‘integer’ default value: ‘800’ 


Controls width of the desired resolution for current application main window. 

Note: there is a contextual resolution in D’Fusion on how this resolution is controlled, and in some context parameters 
in this file are ignored. This is the case for example in a web context, where resolution is controlled by the html page 
directly. 


Parameter : “height’ Type : ‘integer’ default value: ‘600’ 


Controls height of the desired resolution for current application main window. 

Note: there is a contextual resolution in D’ Fusion on how this resolution is controlled, and in some context parameters 
in this file are ignored. This is the case for example in a web context, where resolution is controlled by the html page 
directly. 


Parameter : ‘antialiasing’ Type: ‘integer’ default value: ‘0’ 


This parameter controls the desired antialiasing value for the player. Warning: this setting is a start-up only value, 
which means it cannot be dynamically changed once the player is running. If a .dpd file is used to launch the player, the 
setting will then be used. Otherwise, if loading a scene dynamically, this setting will be ignored. 

Notice also there is a safe management of this value: some computer may not support antialiasing. If initialization of the 
graphic context fails with the given value, it will retry with downgrading this value progressively to 0. (eg, if we fail 
with a value of 4, we will try values of 2, then 1, then 0 if we keep on failing. Error will be reported only if we still 
cannot initialize the context with a value of 0). 


Parameter : “vsync’ Type: ‘boolean’ default value: ‘true’ 

This value controls the vertical synchronization of rendering operations. Most of the time, this parameter is set to true 
in our applications. 

Like the antialiasing value, this parameter is efficient only for the start-up configuration and cannot be changed 
dynamically once the player is loaded. 


Parameter : ‘renderer’ Type : ‘string’ default value: ‘direct3D9’ 


This value controls the renderer used by our graphic engine. As for today, two renderers are available: direct3D9 and 
GL. 
Like antialiasing and vsync, this parameter cannot be changed dynamically and will be used only for start-up. 
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5.2.2 Application 


This node is an optional tag useful in a standalone deployment context. When deploying @Home player, you may 
want to customize your application window. You can specify here some customization of this window. All the 
parameters defined here are used only at start-up and cannot be changed dynamically. 


Application -Parameters 


Parameter : ‘title’ Type : ‘string’ default value: ‘D’ Fusion AR Player’ 
Modifies the title of the player. 


Parameter : ‘icon’ Type: ‘string’ default value: — 


This parameter lets you specify a customized icon for the application. The icon must be a valid image file (.ico for 
example). 


D’Fusion® Augmented Reality 2 project are XML files with ‘.dpd’ extension. 


{The setting tag only applies to the main scene 


If you need to refer an object, it must be declared first ! 
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5.3 Video Input File 


Video inputs are described by configuration files. Lots of parameters permit the user to really adapt the input to 
his needs, but for this quick project let us consider a standard input from a simple webcam. The configuration file 
which we'll call video.xml! looks like this: 


<?xml version="1.0" standalone= "yes" ?> 
<7I> 
<VIDEOCAP> 
<VIDEO_CAPTURE_NUM_DRIVER>0</VIDEO_CAPTURE_NUM_DRIVER> 
<VIDEO_CAPTURE_PIXEL_FORMAT>RGB32</VIDEO_CAPTURE_PIXEL_FORMAT> 
< VIDEO_CAPTURE_WIDTH>640</ VIDEO_CAPTURE_WIDTH> 
< VIDEO_CAPTURE_HEIGHT>480</ VIDEO_CAPTURE_HEIGHT> 
<VIDEO_CAPTURE_DELAY>4</VIDEO_CAPTURE_DELAY> 
<VIDEO_CAPTURE_NB_IMG>30</VIDEO_CAPTURE_NB_IMG> 
<VIDEO_CAPTURE_RATE>15.0</VIDEO_CAPTURE_RATE> 
</VIDEOCAP> 
</TI> 


The main videocapture parameters are listed in the following table : 


Parameter’s name Type Typical Description 
values 
video_capture_num_driver Integer | -1, 0 -1 Indicates the video capture is done through a 
“Num Driver” video file. Otherwise, this number indicates the 
referenced channel corresponding to your camera 
device. 
video_capture_width Integer | 320, 640, 720 | Capture width & Height. 
video_capture_height 
“Capture Size Width & Height” 
video_capture_pixel_format Integer | RGB24, BGR24, | Pixel format for the video capture. Only these 
“Pixel Format” RGB32, mode are available when using D’Fusion 
UYVY422, Computer Vision 
YUY2, GRAY8 
video_capture_rate Double | 25 (PAL), Required frame rate for the capture. The capture 
“Rate” 29.97 (NTSC), | device must fit with the specified frame rate. 
50 (Ueye) 
video_capture_type Integer | 0 0 for Direct Show video capture 
“Type” 1 for Video For Windows video capture 
5 for RTP video capture 
video_capture_interlaced Integer | 0, 1,2 0 for non-interlaced mode, 1 for interlaced mode 
“Interlaced” with the first received line is an even line, 2 for 
interlaced mode with the first received line is an 
odd line. 
video_capture_inverted Integer | 0, 1, 2, 3 0 not flip, 
“Inverted” (0) 1 vertical flip, 
2 horizontal flip, 
3 vertical and horizontal flip. 
video_capture_cpu Integer | -1, 0 CPU on which the video capture thread will run. 
“CPU” -1 no preference; the operating system will 
choose. 
O selects the first. 
1 selects the second. 
2 selects the third. etc... 
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video_capture_delay Integer | 1 Number of video frames between capture and 
“Delay” display. 

When “Most recent frame” widget is checked, 
this additional delay has no effect. 
video_capture_priority Integer | 3 Priority of the capture thread (between -3 and 3) 
“Priority” -3 lowest priority 
3 highest priority 
video_capture_file String | NULL Video file 
“Video File” 
video_capture_friendly_name String NULL Sub-string which has to be found into the device 
“Device List” friendly name to select it. 
video_capture_nb_frame_buffers | Integer | 4 Number of frames in the frame memory. 
“NbFrameBuffers” 
video_capture_shader Integer | 0,1 0 software YUV conversion 
1 hardware YUV conversion 
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6 PLUG-INS CUSTOMIZATION 


Augmented Reality applications may need to use some very specific features dedicated to the environment where 
the application is to be played. You may for example need to use a specific hardware, or use a very special effect 
on your rendering, compute some data from a set of captors, etc. 


Since the playgrounds where you may want to use D’Fusion® Augmented Reality can be so different from each 
other, we designed D’Fusion® Augmented Reality as an open API where you can easily create your plug-ins and 
embed them in your application. 


To create a Plug-in for D’Fusion® Augmented Reality, you just need a couple of simple steps: 


e Create an entry function in your dll project. This entry function must match the plug-in signature to be 
recognized by the engine. This is where you will initialize all your plug-in needs to work smoothly. 


e Refer to a basic set of D’Fusion® headers and link with D’Fusion® Augmented Reality libraries 


e Build your project in the “plugins” folder under the D’Fusion® Player so that is it dynamically loaded on 
start-up 


6.1 Entry Function 


This is main requirement to create your own plug-in. The signature of this function is simply: 


extern "C" PLUGINEXPORT tiError dfusionPluginEntry(darcContext* ctx) | 


As soon as your project exports this function, and is located in the correct folder, it will be considered as a plugin 
for D’Fusion® Augmented Reality. You just need to do then, in this call, all the initialization you may need. For 
example you might create a manager, create some new kind of objects and declare them to the context, register 
some event callbacks, or register some new Lua functions that you will then be able to call directly in your scripts. 


It is your responsibility to manage everything you deal with in your plug-ins, like memory management for 
example. If you initialize a manager, you should always register a callback function in the context closing the 
event to release everything you allocated for example. 


6.2 Using D’Fusion® AR SDK 


As soon as you start making your own plug-ins, you will need to know how to use the D’Fusion® AR SDK. You can 
refer to the D’Fusion® AR - Lua API documentation [03] for a good overview of the SDK. 


One of the most interesting features is the possibility to bind your own functions or objects to Lua. 


REMINDER 


fl A plugin is identified only by its entry function 
{ When you create a plug-in, you must manage completely your own data, including disallocation 


& Binding new functions to Lua is a quick way to customize your applications an efficient way. 
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